Dynamic calibration of lidar sensors

ABSTRACT

A method of calibrating a LiDAR sensor mounted on a vehicle includes storing a reference three-dimensional image acquired by the LiDAR sensor while the LiDAR sensor is in an expected alignment with respect to the vehicle. The reference three-dimensional image includes a first image of a fixed feature on the vehicle. The method further includes, acquiring, using the LiDAR sensor, a three-dimensional image including a second image of the fixed feature, and determining a deviation from the expected alignment of the LiDAR sensor with respect to the vehicle by comparing the second image of the fixed feature in the three-dimensional image to the first image of the fixed feature in the reference three-dimensional image.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/915,563, filed on Oct. 15, 2019, the content of whichis incorporated by reference in its entirety.

The following two U.S. Patent Applications (including this one) arebeing filed concurrently, and the entire disclosure of the otherapplication is incorporated by reference into this application for allpurposes:

-   Application Ser. No. 17/069,727, filed on Oct. 13, 2020, entitled    “CALIBRATION OF LIDAR SENSORS,” and-   Application Ser. No. 17/069,733, filed on Oct. 13, 2020, entitled    “DYNAMIC CALIBRATION OF LIDAR SENSORS.”

BACKGROUND

Three-dimensional sensors can be applied in autonomous vehicles, drones,robotics, security applications, and the like. LiDAR sensors are a typeof three-dimensional sensors that can achieve high angular resolutionsappropriate for such applications. A LiDAR sensor can include one ormore laser sources for emitting laser pulses, and one or more detectorsfor detecting reflected laser pulses. The LiDAR sensor measures the timeit takes for each laser pulse to travel from the LiDAR sensor to anobject within the sensor's field of view, then bounce off the object andreturn to the LiDAR sensor. Based on the time of flight of the laserpulse, the LiDAR sensor determines how far away the object is from theLiDAR sensor. By scanning across a scene, a three-dimensional image ofthe scene may be obtained.

For accurate measurements, the orientation of the optical axis of aLiDAR sensor may need to be calibrated with respect to some mechanicaldatum point, such as mounting holes on a case of the LiDAR sensor.Additionally, when mounted in a vehicle, the position and theorientation of the LiDAR sensor may need to be calibrated with respectto the vehicle. Such calibrations can be performed, for example, in amanufacturer's plant. In the event of a crash or other mechanicaldisturbance to the LiDAR sensor, its calibration with respect to eitherthe case or the vehicle might change. Thus, it may be desirable to beable to detect a loss of calibration accuracy and to correct thecalibration, so as to ensure safe and accurate long term operation of aLiDAR sensor.

SUMMARY

According to some embodiments, a method of calibrating a LiDAR sensormounted on a vehicle includes positioning the vehicle at a distance froma target. The target includes a planar mirror and features surroundingthe mirror. The optical axis of the mirror is substantially horizontal.The vehicle is positioned and oriented relative to the mirror so that anoptical axis of the LiDAR sensor is nominally parallel to the opticalaxis of the mirror, and the target is nominally centered at a field ofview of the LiDAR sensor. The method further includes acquiring, usingthe LiDAR sensor, a three-dimensional image of the target. Thethree-dimensional image of the target includes images of the features ofthe target and a mirror image of the vehicle formed by the mirror. Themethod further includes determining a deviation from an expectedalignment of the LiDAR sensor with respect to the vehicle by analyzingthe images of the features and the mirror image of the vehicle in thethree-dimensional image of the target.

According to some embodiments, a method of calibrating a LiDAR sensormounted on a vehicle includes storing a reference three-dimensionalimage acquired by the LiDAR sensor while the LiDAR sensor is in anexpected alignment with respect to the vehicle. The referencethree-dimensional image includes a first image of a fixed feature on thevehicle. The method further includes, acquiring, using the LiDAR sensor,a three-dimensional image including a second image of the fixed feature,and determining a deviation from the expected alignment of the LiDARsensor with respect to the vehicle by comparing the second image of thefixed feature in the three-dimensional image to the first image of thefixed feature in the reference three-dimensional image.

According to some embodiments, a method of calibrating a LiDAR sensormounted on a vehicle includes acquiring, using the LiDAR sensor whilethe vehicle is traveling on a road with fixed road features, one or morethree-dimensional images. Each of the one or more three-dimensionalimages includes images of the road features. The method further includesanalyzing a spatial relationship between the images of the road featuresin the one or more three-dimensional images and an orientation of afield of view of the LiDAR sensor, and determining a deviation from anexpected alignment of the LiDAR sensor with respect to the vehicle basedon the spatial relationship between the images of the road features andthe field of view of the LiDAR sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary LiDAR sensor for three-dimensionalimaging according to some embodiments.

FIG. 2A shows schematically a LiDAR sensor mounted on a vehicleaccording to some embodiments.

FIG. 2B illustrates a vehicle coordinate system according to someembodiments.

FIG. 2C illustrates a LiDAR coordinate system according to someembodiments.

FIGS. 3A and 3B show schematically a mounting mechanism that can be usedto mount a LiDAR sensor on a vehicle according to some embodiments.

FIGS. 4A-4C illustrate an exemplary calibration setup for a LiDAR sensormounted on a vehicle according to some embodiments.

FIGS. 5A-5C and FIGS. 6A-6D illustrate schematically what a LiDAR sensormay see under various alignment conditions according to someembodiments.

FIG. 7 shows a simplified flowchart illustrating a method of calibratinga LiDAR sensor mounted on a vehicle using a target with an embeddedmirror according to some embodiments.

FIG. 8 illustrates a LiDAR sensor mounted behind the front windshield ofthe vehicle that includes a mask according to some embodiments.

FIGS. 9A and 9B show schematically side views of the windshield with themask illustrated in FIG. 8.

FIG. 10A-10C illustrate schematically some examples of effects of themask illustrated in FIG. 8 on the effective field of view of the LiDARsensor under various alignment conditions according to some embodiments.

FIGS. 10D-10F illustrate schematically some exemplary images of certainfeatures of the vehicle acquired by the LiDAR sensor according to someembodiments.

FIG. 11 shows a simplified flowchart illustrating a method ofcalibrating a LiDAR sensor mounted on a vehicle using features on thevehicle according to some embodiments.

FIG. 12A-12B and FIGS. 13A-13C illustrate a method of dynamiccalibration of a LiDAR sensor mounted on a vehicle using lane markingson a road according to some embodiments.

FIG. 14 shows a simplified flowchart illustrating a method ofcalibrating a LiDAR sensor mounted on a vehicle using road featuresaccording to some embodiments.

DETAILED DESCRIPTION

According to some embodiments, methods of calibrating a LiDAR sensormounted on a vehicle are provided. The calibration may not requirereturning the vehicle with the LiDAR sensor mounted thereon to amanufacturer's plant or a repair shop. Calibrations can be performedperiodically or continuously while the vehicle is parked or even duringdriving.

FIG. 1 illustrates an exemplary LiDAR sensor 100 for three-dimensionalimaging according to some embodiments. The LiDAR sensor 100 includes anemission lens 130 and a receiving lens 140. The LiDAR sensor 100includes a light source 110 a disposed substantially in a back focalplane of the emission lens 130. The light source 110 a is operative toemit a light pulse 120 from a respective emission location in the backfocal plane of the emission lens 130. The emission lens 130 isconfigured to collimate and direct the light pulse 120 toward an object150 located in front of the LiDAR sensor 100. For a given emissionlocation of the light source 110 a, the collimated light pulse 120′ isdirected at a corresponding angle toward the object 150.

A portion 122 of the collimated light pulse 120′ is reflected off of theobject 150 toward the receiving lens 140. The receiving lens 140 isconfigured to focus the portion 122′ of the light pulse reflected off ofthe object 150 onto a corresponding detection location in the focalplane of the receiving lens 140. The LiDAR sensor 100 further includes adetector 160 a disposed substantially at the focal plane of thereceiving lens 140. The detector 160 a is configured to receive anddetect the portion 122′ of the light pulse 120 reflected off of theobject at the corresponding detection location. The correspondingdetection location of the detector 160 a is optically conjugate with therespective emission location of the light source 110 a.

The light pulse 120 may be of a short duration, for example, 10 ns pulsewidth. The LiDAR sensor 100 further includes a processor 190 coupled tothe light source 110 a and the detector 160 a. The processor 190 isconfigured to determine a time of flight (TOF) of the light pulse 120from emission to detection. Since the light pulse 120 travels at thespeed of light, a distance between the LiDAR sensor 100 and the object150 may be determined based on the determined time of flight.

One way of scanning the laser beam 120′ across a FOV is to move thelight source 110 a laterally relative to the emission lens 130 in theback focal plane of the emission lens 130. For example, the light source110 a may be raster scanned to a plurality of emission locations in theback focal plane of the emission lens 130 as illustrated in FIG. 1. Thelight source 110 a may emit a plurality of light pulses at the pluralityof emission locations. Each light pulse emitted at a respective emissionlocation is collimated by the emission lens 130 and directed at arespective angle toward the object 150, and impinges at a correspondingpoint on the surface of the object 150. Thus, as the light source 110 ais raster scanned within a certain area in the back focal plane of theemission lens 130, a corresponding object area on the object 150 isscanned. The detector 160 a may be raster scanned to be positioned at aplurality of corresponding detection locations in the focal plane of thereceiving lens 140, as illustrated in FIG. 1. The scanning of thedetector 160 a is typically performed synchronously with the scanning ofthe light source 110 a, so that the detector 160 a and the light source110 a are always optically conjugate with each other at any given time.

By determining the time of flight for each light pulse emitted at arespective emission location, the distance from the LiDAR sensor 100 toeach corresponding point on the surface of the object 150 may bedetermined. In some embodiments, the processor 190 is coupled with aposition encoder that detects the position of the light source 110 a ateach emission location. Based on the emission location, the angle of thecollimated light pulse 120′ may be determined. The X-Y coordinate of thecorresponding point on the surface of the object 150 may be determinedbased on the angle and the distance to the LiDAR sensor 100. Thus, athree-dimensional image of the object 150 may be constructed based onthe measured distances from the LiDAR sensor 100 to various points onthe surface of the object 150. In some embodiments, thethree-dimensional image may be represented as a point cloud, i.e., a setof X, Y, and Z coordinates of the points on the surface of the object150.

In some embodiments, the intensity of the return light pulse 122′ ismeasured and used to adjust the power of subsequent light pulses fromthe same emission point, in order to prevent saturation of the detector,improve eye-safety, or reduce overall power consumption. The power ofthe light pulse may be varied by varying the duration of the lightpulse, the voltage or current applied to the laser, or the charge storedin a capacitor used to power the laser. In the latter case, the chargestored in the capacitor may be varied by varying the charging time,charging voltage, or charging current to the capacitor. In someembodiments, the reflectivity, as determined by the intensity of thedetected pulse, may also be used to add another dimension to the image.For example, the image may contain X, Y, and Z coordinates, as well asreflectivity (or brightness).

The angular field of view (AFOV) of the LiDAR sensor 100 may beestimated based on the scanning range of the light source 110 a and thefocal length of the emission lens 130 as,

${{AFOV} = {2\mspace{11mu} {\tan^{- 1}( \frac{h}{2f} )}}},$

where h is scan range of the light source 110 a along certain direction,and f is the focal length of the emission lens 130. For a given scanrange h, shorter focal lengths would produce wider AFOVs. For a givenfocal length f, larger scan ranges would produce wider AFOVs. In someembodiments, the LiDAR sensor 100 may include multiple light sourcesdisposed as an array at the back focal plane of the emission lens 130,so that a larger total AFOV may be achieved while keeping the scan rangeof each individual light source relatively small. Accordingly, the LiDARsensor 100 may include multiple detectors disposed as an array at thefocal plane of the receiving lens 140, each detector being conjugatewith a respective light source. For example, the LiDAR sensor 100 mayinclude a second light source 110 b and a second detector 160 b, asillustrated in FIG. 1. In other embodiments, the LiDAR sensor 100 mayinclude four light sources and four detectors, or eight light sourcesand eight detectors. In one embodiment, the LiDAR sensor 100 may includeeight light sources arranged as a 4×2 array and eight detectors arrangedas a 4×2 array, so that the LiDAR sensor 100 may have a wider AFOV inthe horizontal direction than its AFOV in the vertical direction.According to various embodiments, the total AFOV of the LiDAR sensor 100may range from about 5 degrees to about 15 degrees, or from about 15degrees to about 45 degrees, or from about 45 degrees to about 120degrees, depending on the focal length of the emission lens, the scanrange of each light source, and the number of light sources.

The light source 110 a may be configured to emit light pulses in theultraviolet, visible, or near infrared wavelength ranges. The energy ofeach light pulse may be in the order of microjoules, which is normallyconsidered to be eye-safe for repetition rates in the KHz range. Forlight sources operating in wavelengths greater than about 1500 nm, theenergy levels could be higher as the eye does not focus at thosewavelengths. The detector 160 a may comprise a silicon avalanchephotodiode, a photomultiplier, a PIN diode, or other semiconductorsensors.

When a LiDAR sensor, such as the LiDAR sensor 100 illustrated in FIG. 1,is used for obstacle detection for autonomous vehicles, the position andorientation of the LiDAR sensor with respect to the vehicle may need tobe accurately known in order to accurately determine the location of anobstacle relative to the vehicle. FIG. 2A shows schematically a LiDARsensor 210 mounted on a vehicle 220. For example, the LiDAR sensor 210can be mounted at the top center behind the windshield of the vehicle220. The LiDAR sensor 210 can be characterized by an optical axis 250(e.g., the optical axis of an emission lens or a receiving lens). Thevehicle 220 can have a longitudinal axis 240. It may be advantageous toalign the optical axis 250 of the LiDAR sensor 210 with the longitudinalaxis 240 of the vehicle 220, so that the LiDAR sensor 210 looks straightahead toward the direction the vehicle 220 is heading.

Assume that the LiDAR sensor 210 is initially calibrated according tothis alignment condition. If the LiDAR sensor's orientation is shifted(e.g., turned toward the left) due to some mechanical disturbances, thenthe optical axis 250 of the LiDAR sensor 210 is no longer aligned withthe longitudinal axis 240 of the vehicle 220, as illustrated in FIG. 2A.The mis-alignment may result in inaccurate measurements of a position ofan obstacle (e.g., a person 260) relative to the vehicle 220. Thus,re-calibration of the LiDAR sensor 210 may be required.

FIG. 2B illustrates a vehicle coordinate system. The vehicle coordinatesystem can have three degrees of translational freedom, which can berepresented, for example, by the x, y, and z coordinates. The vehiclecoordinate system can have three degrees of rotational freedom, whichcan be represented, for example, by the roll, pitch, and yaw anglesabout the x-axis, the y-axis, and the z-axis, respectively. For example,the x-axis can be along a longitudinal axis of the vehicle 220; thez-axis can be along the vertical direction; and the y-axis can be alongthe lateral direction. The position and the orientation of the LiDARsensor 210 relative to the vehicle 220 can be characterized by the (x,y, z, roll, pitch, yaw) coordinates in the vehicle coordinate system.

FIG. 2C illustrates a LiDAR coordinate system. The LiDAR coordinatesystem can also have three degrees of translational freedom (e.g.,represented by the X, Y, and Z coordinates) and three degrees ofrotational freedom (e.g., represented by the roll, pitch, and yaw anglesabout the X-axis, the Y-axis, and the Z-axis, respectively). Forexample, the X-axis can be along an optical axis of the LiDAR sensor210; the Z-axis can be along the nominal vertical direction; and theY-axis can be along a direction orthogonal to the X-axis and the Z-axis.

The raw data of a point cloud acquired by the LiDAR sensor 210 can be inthe LiDAR coordinate system. To determine the location of an obstaclerelative to the vehicle 220, the point cloud data can be transformedinto the vehicle coordinate system, if the position and the orientationof the LiDAR sensor 210 in the vehicle coordinate system is known. Thetransformation from the LiDAR coordinate system into the vehiclecoordinate system may be referred herein as calibration of the LiDARsensor 210 with respect to the vehicle 220.

According to some embodiments, the LiDAR sensor 220 can be mounted onthe vehicle 220 so that the LiDAR sensor 210 is nominally aligned withrespect to the vehicle 220. For example, the LiDAR sensor 210 can bemounted on the vehicle 220 such that the X-axis of the LiDAR coordinatesystem (e.g., along the optical axis of the LiDAR sensor 210) isnominally aligned with the x-axis of the vehicle coordinate system(e.g., along the longitudinal axis of the vehicle); the Y-axis of theLiDAR coordinate system is nominally aligned with the y-axis of thevehicle coordinate system; and the Z-axis of the LiDAR coordinate systemis nominally aligned with the z-axis of the vehicle coordinate system.Thus, the roll angle, the pitch angle, and the yaw angle in the vehiclecoordinate system are all approximately zero. A calibration may berequired to compensate for any residual deviations from the nominalalignment to a sufficient accuracy. For example, it may be desirable tocalibrate the LiDAR sensor 210 to a translational accuracy of 2 cm alongeach of the x-, y-, and z-axes, and a rotational accuracy of 0.1 degreesfor each of the roll, pitch, and yaw angles.

FIGS. 3A and 3B show schematically a mounting mechanism that can be usedto mount a LiDAR sensor on a vehicle according to some embodiments. TheLiDAR sensor can have an outer housing 310 with three mounting holes 320(e.g., on the top surface of the outer housing 310), as illustrated inFIG. 3A. A bracket 330 can have three holes 340 that match with themounting holes 320 on the outer housing 310, so that the LiDAR sensorcan be attached to the bracket 330 with a fixed orientation, asillustrated in FIG. 3B. The bracket 330 can be attached to the vehicle(e.g., to the roof of the interior compartment), using similar mountingholes (not shown) for proper alignment.

A LiDAR sensor can be pre-calibrated in a manufacturer's plant tocorrect for any residual mis-alignment of the LiDAR sensor with respectto the vehicle. During the operation of the vehicle over time, theoptical components of the LiDAR sensor can be shifted relative to thehousing 310, or the housing 310 of the LiDAR sensor can be shiftedrelative to the mounting bracket 330 and/or to the vehicle. This mayhappen, for example, due to tear and wear of the internal mechanism ofthe LiDAR sensor, a collision or vibrations of the vehicle,mis-alignment of the tires, aging of the vehicle's suspension, and thelike. Thus, over time, the calibration can become inaccurate and a newcalibration may be required. According to various embodiments, LiDARcalibrations can be performed periodically using a target with anembedded mirror, or can be performed periodically or continuously usingfixed features on the vehicle or fixed road features, as described inmore detail below.

a. Calibration of Lidar Sensors Using a Target with an Embedded Mirror

FIGS. 4A-4C illustrate an exemplary calibration setup for a LiDAR sensormounted on a vehicle using a target with an embedded mirror according tosome embodiments. The LiDAR sensor 410 is shown as mounted on behind thefront windshield of the vehicle 420 in this example. This is however notrequired. For example, the LiDAR sensor 410 can be mounted on otherlocations on the vehicle 420, such as on the front bumper, the rearwindshield, the rear bumper, and the like. A target 440 is placed atcertain distance D in front of the LiDAR sensor 410. The target 440includes a planar mirror 430 and identifiable features 490 surroundingthe mirror 430.

Referring to FIG. 4A, the target 440 can be mounted on a stand (notshown), so that the mirror 430 is nominally vertical. Thus, an opticalaxis 432 of the mirror 430, which is normal to the surface of the mirror430, can be nominally horizontal. The vehicle 420 is positioned andoriented relative to the mirror 430 so that an optical axis 412 of theLiDAR sensor is nominally parallel to the optical axis 432 of the mirror430, and the target 440 is nominally centered at a field of view of theLiDAR sensor 410. In the cases in which the LiDAR sensor 410 is mountedbehind the front windshield of the vehicle 420 (e.g., as illustrated inFIG. 4A), the vehicle 420 can be positioned so that a longitudinal axis422 of the vehicle 420 is nominally parallel to the optical axis 432 ofthe mirror 430.

Referring to FIG. 4B, the LiDAR sensor 410 can acquire athree-dimensional image of the target 440 by scanning across the target440. The three-dimensional image of the target can include images of thefeatures 490 surrounding the mirror 430, and a mirror image of thevehicle 420 formed by the mirror 430. For example, the LiDAR sensor 410can emit a laser pulse 450 (or 450 a) directed toward the mirror 430. Areflected laser pulse 450 b can be directed toward certain part of thevehicle (e.g., the license plate at the front bumper), which can in turnbe reflected back toward the mirror 430 (indicated as 450 c in FIG. 4B),and return back to LiDAR sensor 410 (indicated as 450 d in FIG. 4B).Thus, the LiDAR sensor 410 can effectively acquire a three-dimensionalmirror image of the vehicle 420, as illustrated in FIG. 4C.

According to some embodiments, the distance D between the target 440 andthe vehicle 420 can be made large enough to meet desired accuracy. Forexample, the distance D can be about 3 m. The size of the mirror 430 canbe made large enough so that the entire vehicle 420 is visible, althoughthis is not required. For example, the size of the mirror 430 can beabout 1 m tall and about 2 m wide. The size of the target 440 can beabout 2 m tall and about 3 m wide (e.g., the edge of the target 440 withfeatures 490 can have a 0.5 m width).

FIG. 5A illustrates schematically what the LiDAR sensor 410 may see whenthe vehicle 420 is correctly aligned with respect to the target 440(e.g., the longitudinal axis of the vehicle 420 is perpendicular to thesurface of the mirror 430), and the LiDAR sensor 410 is correctlyaligned with respect to the vehicle 420 (e.g., the optical axis of theLiDAR sensor 410 is parallel to the longitudinal axis of the vehicle420). Assume also that the vehicle 420 is laterally centered withrespect to the width of the mirror 430, and the LiDAR sensor 410 islaterally centered with respect to the vehicle 420. In such cases, themirror image 420′ of the vehicle 420 as seen from the LiDAR sensor 410may appear symmetrical, and may not have any roll angle, pitch angle, oryaw angle. The target 440 may also appear symmetrical with respect tothe LiDAR sensor's field of view 414 (e.g., the lateral margin d fromthe left edge of the target 440 to the left border of the field of view414 is about the same as the lateral margin d from the right edge of thetarget 440 to the right border of the field of view 414), and may nothave any roll angle, pitch angle, or yaw angle.

FIG. 5B illustrates schematically what the LiDAR sensor 410 may see whenthe LiDAR sensor 410 is correctly aligned with respect to the vehicle420, but the vehicle 420 is misaligned with respect to the target 440(e.g., the longitudinal axis of the vehicle 420 has a finite yaw anglewith respect to the optical axis of the mirror 430). In such cases, thetarget 440 is no longer centered in the LiDAR's field of view 414 (e.g.,laterally shifted toward the right). The amount of shift (e.g., thedifference between the new margin D and the nominal margin d) can relateto the amount of mis-alignment. Thus, it may be possible to determinethe amount of mis-orientation (e.g., the yaw error) of the vehicle 420with respect to the target 440 based on the amount of shift. Inaddition, the mirror image 420′ of the vehicle 420 as seen by the LiDARsensor 410 also appears to have a finite yaw angle (note that the amountof the rotation is somewhat exaggerated in FIG. 5B). Thus, the LiDARsensor 410 can also determine the amount of mis-orientation of thevehicle 420 with respect to the target 440 by measuring the distances ofcertain features of the vehicle 420. For example, as illustrated in FIG.5B, the LiDAR sensor 410 can measure that one headlamp (e.g., the leftheadlamp) is farther from the LiDAR sensor 410 than the other headlamp(e.g., the right headlamp), and thus calculate the yaw angle of thevehicle 420.

FIG. 5C illustrates schematically what the LiDAR sensor 410 may see whenthe vehicle 420 is correctly aligned with respect to the target 440(e.g., the longitudinal axis of the vehicle 420 is perpendicular to thesurface of the mirror 430), but the LiDAR sensor 410 is not properlyaligned with respect to the vehicle 420 (e.g., the LiDAR sensor 410looks to the left instead of looking straight ahead). In such cases, themirror image 420′ of the vehicle 420 may appear to be symmetric, but thetarget 440 may be laterally shifted (e.g., to the right) with respect tothe LiDAR's field of view 414. The LiDAR sensor 410 can determine theamount of mis-orientation (e.g., the yaw error) of the LiDAR sensor 410based on the amount of the shift (e.g., the difference between the newmargin D and the nominal margin d).

FIG. 6A illustrates schematically what the LiDAR sensor 410 may see whenthe LiDAR sensor 410 is correctly aligned with respect to the vehicle420, but the vehicle 420 has a pitch error with respect to the target440. In such cases, the target 440 is no longer centered in the LiDAR'sfield of view 414 (e.g., vertically shifted downward). The amount ofvertical shift can relate to the amount of pitch error. Thus, it may bepossible to determine the amount of pitch error of the vehicle 420 withrespect to the target 440 based on the amount of vertical shift. Inaddition, the mirror image 420′ of the vehicle 420 as seen by the LiDARsensor 410 also appears to have a finite pitch angle (note that theamount of the rotation is somewhat exaggerated in FIG. 6A). Thus, theLiDAR sensor 410 can also determine the amount of pitch error of thevehicle 420 with respect to the target 440 by measuring the distances ofcertain features of the vehicle 420. For example, as illustrated in FIG.6A, the LiDAR sensor 410 can measure that the roof of the vehicle 420 istilted, and can calculate the pitch error based on the amount of tilt.

FIG. 6B illustrates schematically what the LiDAR sensor 410 may see whenthe vehicle 420 is correctly aligned with respect to the target 440, butthe LiDAR sensor 410 has a pitch error with respect to the vehicle 420(e.g., the LiDAR sensor 410 looks upward instead of looking straightahead). In such cases, the mirror image 420′ of the vehicle 420 mayappear to have no pitch angle, but the target 440 may be verticallyshifted (e.g., downward) with respect to the LiDAR's field of view 414.The LiDAR sensor 410 can determine the amount of pitch error of theLiDAR sensor 410 based on the amount of the vertical shift.

FIG. 6C illustrates schematically what the LiDAR sensor 410 may see whenthe LiDAR sensor 410 is correctly aligned with respect to the vehicle420, but the vehicle 420 has a roll error with respect to the target440. In such cases, the mirror image 420′ of the vehicle 420 as seen bythe LiDAR sensor 410 appears to have no roll angle, but the target 440can appear to have a finite roll angle with respect to the LiDAR's fieldof view 414. It may be possible to determine the amount of roll error ofthe vehicle 420 with respect to the target 440 based on the amount ofthe roll angle of the target 440 with respect to the LiDAR sensor'sfield of view 414.

FIG. 6D illustrates schematically what the LiDAR sensor 410 may see whenthe vehicle 420 is correctly aligned with respect to the target 440, butthe LiDAR sensor 410 has a roll error with respect to the vehicle 420.In such cases, both the mirror image 420′ of the vehicle 420 and thetarget 440 may appear to have a finite roll angle with respect to theLiDAR's field of view 414. The LiDAR sensor 410 can determine the amountof roll error of the LiDAR sensor 410 based on the amount of the rollrotation.

Therefore, the LiDAR sensor 410 (or a computing unit of the vehicle 420)can determine a deviation from an expected alignment (e.g., an initialalignment performed in a manufacturer's plant) of the LiDAR sensor 410with respect to the vehicle 420 by analyzing three-dimensional imageacquired by the LiDAR sensor 410, which includes the images of thefeatures 490 on the target and the mirror image 420′ of the vehicle 420.The deviation from the expected alignment can include yaw error, rollerror, pitch error, and translational errors (e.g., δx, δy, δz).

According to some embodiments, determining the deviation from theexpected alignment of the LiDAR sensor 410 with respect to the vehicle420 can include the following steps. A position and an orientation ofthe LiDAR sensor 410 relative to the target 440 can be determined basedon the images of the features 490 on the target 440. A position and anorientation of the LiDAR sensor 410 relative to the mirror image 420′ ofthe vehicle 420 can be determined based on the mirror image 420′ ofvehicle. Then, a transformation from a LiDAR coordinate system into avehicle coordinate system can be determined based on: (i) the positionand the orientation of the LiDAR sensor 410 relative to the target 440,and (ii) the position and the orientation of the LiDAR sensor 410relative to the mirror image 420′ of the vehicle 420.

In some embodiments, a computing unit can store a reference image. Forexample, the reference image can be an image acquired by the LiDARsensor 410 just after an initial alignment has been performed at themanufacturer's plant, or can be a simulated image for the expectedalignment. During re-calibration, the computing unit can compare thethree-dimensional image acquired by the LiDAR sensor 410 to thereference image, and perform a multi-variable minimization (e.g., usinga gradient descent or other algorithms) to determine a transformationmatrix so that the acquired three-dimensional image most closely matchesthe reference image. The deviation from the expected alignment (e.g.,yaw error, roll error, pitch error, δx, δy, and δz) can then be derivedfrom the transformation matrix.

The following exemplary methods may be used to determine therelationship of the LiDAR sensor 410 to the vehicle 420, in as many assix degrees of freedom according to various embodiments. Other methodsand techniques may also be used by those skilled in the arts. In thebelow descriptions of the exemplary embodiments, the following notationsand terminologies will be used. L_(t) denotes a matrix describing therelationship of the LiDAR sensor 410 to the target 440, C_(t) denotes amatrix describing the relationship of the vehicle 420 to the target 440,L_(C) denotes a matrix describing the relationship of the LiDAR sensor410 to the vehicle 420 (for correcting any mis-calibration of the LiDARsensor 410), M denotes a mirror transformation matrix, and L_(mC)denotes a matrix describing the relationship of the LiDAR sensor 410 tothe vehicle 420 as the LiDAR sensor 410 sees in the mirror 430.

In some embodiments, the LiDAR sensor 410 can establish its positional(x, y, z) relationship to the target features 490 around the mirror 430by triangulating the distance from at least three target features 490(e.g., similar to how a GPS receiver triangulates its position relativeto the GPS satellites). The rotational relationships of pitch, roll, andyaw can be determined by measuring the location of at least three targetfeatures 490 (which could be the same target features used for x, y, andz). Thus, the matrix describing the relationship of the LiDAR sensor 410to the target 440 L_(t) can be established. The matrix L_(t) is a 4×4matrix that defines x, y, and z position, as well as pitch, roll, andyaw.

Next, the relationship of the vehicle 420 to the target 440 can beestablished by a similar procedure: by measuring the distance to certainfeatures of the vehicle 420 as seen in the mirror 430 to triangulate itsposition relative to the target 440, and by measuring location offeatures on the vehicle 420 relative to the mirror 430 to determine thepitch, roll, and yaw of the vehicle 420. Thus, the matrix describing therelationship of the vehicle 420 to the target 440 C_(t) can beestablished. The matrix C_(t) is a 4×4 matrix that defines x, y, and zposition, as well as pitch, roll, and yaw. The relationship of the LiDARsensor 410 to the vehicle 420 can then be defined by L_(C)=(C_(t)⁻¹)·L_(t).

According to some embodiments, the LiDAR sensor 410 can determine itsposition relative to the target 440 as described above to establish thematrix L_(t). The LiDAR sensor 410 can also determine the relationshipof the mirror image of the vehicle 420′ to the LiDAR sensor 410 as seenthrough the mirror 430 to establish the matrix L_(mC). The relationshipof the LiDAR sensor 410 to the vehicle 420 can then be determined bymatrix multiplication as,

L _(C) =L _(t) ·M(L _(t) ⁻¹)L _(mC) ·M.

According to some embodiments, the image of the vehicle 420′ as seen inthe mirror 430 and the image of the target 440 as acquired by the LiDARsensor 410 are compared to a stored reference image. The reference imagecan be either a simulated image or an image taken during factorycalibration. A minimization technique (e.g., using a gradient descentalgorithm) can then be used to determine the transform parameters (δx,δy, δz, pitch error, roll error, and yaw error) that minimizes thedifference between the currently acquired image and the stored referenceimage. In this process, either the currently acquired image can betransformed to match the stored reference image, or the stored referenceimage can be transformed to match the currently acquired image. Thetransform parameters can represent the difference between the currentLiDAR position and the ideal (or factory calibrated) LiDAR positionrelative to the vehicle 420.

Once the relationship of the LiDAR 410 to the vehicle 420 is determined,any discrepancy of this relationship from the current LiDAR calibrationcan be used to correct the LiDAR calibration.

In some embodiments, one or more distance sensors can be used todetermine the location and the orientation of the vehicle 420 relativeto the target 440 (e.g., in a world coordinate system). For example, byplacing two or three distance sensors in the pavement under the vehicle420, the pitch, roll, and z-coordinate (height) of the vehicle can bedetermined. By using six distance sensors, all degrees of freedom (x, y,z, pitch, roll, and yaw) can be determined. The distance sensors can beultrasonic sensors or laser sensors.

An example of using distance sensors to determine the position and theorientation of the vehicle 420 is illustrated in FIG. 5A. In thisexample, four distance sensors 590 can be positioned in appropriatelocations around the vehicle 420. For example, each distance sensor 590can be located in the vicinity of a respective wheel of the vehicle 420.The distance sensor 590 can send an ultrasonic pulse or a laser pulsetoward the wheel and measure the return pulse, thereby estimating thedistance from the wheel. Thus, the y-coordinate and the yaw angle of thevehicle 420 can be determined. In some embodiments, only two distancesensors 590 on one side of the vehicle (either on the driver side or thepassenger side) may be required. Another distance sensor 592 can bepositioned in front the vehicle 420 to determine the x-coordinate of thevehicle 420.

According to some embodiments, additional corrections or calibrationscan be made. For example, windshield distortion effects can be measuredand corrected. Windshield distortion correction may be necessary everytime the windshield is replaced.

FIG. 7 shows a simplified flowchart illustrating a method 700 ofcalibrating a LiDAR sensor mounted on a vehicle using a target with anembedded mirror according to some embodiments.

The method 700 includes, at 702, positioning the vehicle at a distancefrom the target. The target includes a planar mirror and featuressurrounding the mirror. The optical axis of the mirror is substantiallyhorizontal. The vehicle is positioned and oriented relative to themirror so that an optical axis of the LiDAR sensor is nominally parallelto the optical axis of the mirror, and the target is nominally centeredat a field of view of the LiDAR sensor.

The method 700 further includes, at 704, acquiring, using the LiDARsensor, a three-dimensional image of the target. The three-dimensionalimage of the target includes images of the features of the target and amirror image of the vehicle formed by the mirror.

The method 700 further includes, at 706, determining a deviation from anexpected alignment of the LiDAR sensor with respect to the vehicle byanalyzing the images of the features and the mirror image of the vehiclein the three-dimensional image of the target.

In some embodiments, the method 700 further includes, at 708,re-calibrating the LiDAR sensor with respect to the vehicle based on thedeviation from the expected alignment of the LiDAR sensor with respectto the vehicle.

In some embodiments, the method 700 further includes determining thatthe deviation from the expected alignment of the LiDAR sensor exceeds athreshold, and providing an alert in response to determining that thedeviation from the expected alignment of the LiDAR sensor exceeds thethreshold.

In some embodiments, the field of view of the LiDAR sensor is less than180 degrees in a horizontal direction.

In some embodiments, determining the deviation from the expectedalignment of the LiDAR sensor with respect to the vehicle can include:determining a position and an orientation of the LiDAR sensor relativeto the target based on the images of the features; determining aposition and an orientation of the LiDAR sensor relative to the mirrorimage of the vehicle based on the mirror image of vehicle; anddetermining a transformation from a LiDAR coordinate system into avehicle coordinate system based on: (i) the position and the orientationof the LiDAR sensor relative to the target, and (ii) the position andthe orientation of the LiDAR sensor relative to the mirror image of thevehicle.

In some embodiments, determining the deviation from the expectedalignment of the LiDAR sensor with respect to the vehicle can include:storing a reference matrix relating to an expected relationship betweenthe LiDAR sensor and the vehicle; determining a matrix relating to acurrent relationship between the LiDAR sensor and the vehicle; anddetermining the deviation from the expected alignment of the LiDARsensor with respect to the vehicle by comparing the matrix to thereference matrix. In some embodiments, the method 700 further includesre-calibrating the LiDAR sensor with respect to the vehicle based on thematrix relating to the current relationship between the LiDAR sensor andthe vehicle.

It should be appreciated that the specific steps illustrated in FIG. 7provide a particular method of calibrating a LiDAR sensor according tosome embodiments. Other sequences of steps may also be performedaccording to alternative embodiments. For example, alternativeembodiments of the present invention may perform the steps outlinedabove in a different order. Moreover, the individual steps illustratedin FIG. 7 may include multiple sub-steps that may be performed invarious sequences as appropriate to the individual step. Furthermore,additional steps may be added and some steps may be removed depending onthe particular applications. One of ordinary skill in the art wouldrecognize many variations, modifications, and alternatives.

B. Dynamic Calibration of LiDAR Sensors Using Features on a Vehicle

It may be desirable to have a method of checking the calibration of aLiDAR sensor with respect to a vehicle periodically or continuouslywhile the vehicle is parked or even during driving. According to someembodiments, calibration of the LiDAR sensor can be performed usingfeatures on the vehicle, so that calibration can be performed duringnormal operation of the vehicle. Such methods are referred to herein asdynamic calibration.

FIG. 8 illustrates an example of dynamic calibration of a LiDAR sensorusing a mask attached to a windshield according to some embodiments. Asillustrated, the LiDAR sensor 810 (not visible in FIG. 8) is mountedbehind the front windshield 840 of the vehicle 850. A mask 820 isattached to the windshield 840 in an area right in front of the LiDARsensor 810. Assuming that the operating wavelength of the LiDAR sensor810 is in the IR wavelength range, the mask 820 can be configured toblock infrared (IR) light (e.g., optically opaque to infrared light). Insome embodiments, the mask 820 can be in an annular shape surrounding anIR-transparent portion 830 of the windshield 840. The geometry and thesize of the IR-blocking mask 820 can be made so that a perimeter aroundthe edge of the field of view of the LiDAR sensor 810 is blocked.

FIGS. 9A and 9B show schematically side views of the windshield 840 withthe IR-blocking mask 820 according to some embodiments. TheIR-transparent portion 830 of the windshield 840 is surrounded by theIR-blocking mask 820. The IR-transparent portion 830 is slightly smallerthan the field of view 910 of the LiDAR sensor 810. Referring to FIG.9A, under a correct alignment (e.g., an expected alignment), the LiDARsensor 810 can be positioned behind the windshield 840 so that its fieldof view 910 is centered about the IR-transparent portion 830. Thus, anedge of the field of view 910 is blocked on all four sides (the sideview of FIG. 9A shows the upper and lower sides).

FIG. 9B shows an example when the LiDAR sensor 810 is shifted from thecorrect alignment position (e.g., tilted upward). As a result, theIR-transparent portion 830 is no longer centered about the field of view830 of the LiDAR sensor 810. Thus, a larger portion of its field of view910 can be blocked on the upper side than on the lower side.

FIG. 10A-10C illustrate schematically some examples of effects of theIR-blocking mask 820 illustrated in FIG. 8 under various alignmentconditions according to some embodiments. FIG. 10A illustrates anexample in which the LiDAR sensor 810 is correctly aligned. TheIR-transparent portion 830 of the windshield 840 is centered about thefield of view 910 of the LiDAR sensor 810. Thus, the edge of the fieldof view 910 that is blocked by the IR-blocking mask 820 (the grey area)has the about the same width on all four sides.

FIG. 10B illustrates an example in which the LiDAR sensor 810 is shiftedtoward the right from the correct alignment position (e.g., the LiDARsensor 810 has a yaw error). As a result, the IR-transparent portion 830of the windshield 840 is shifted toward the left with respect to thefield of view 910 of the LiDAR sensor 810. Thus, a larger portion of thefield of view 910 is blocked by the IR-blocking mask 820 on the rightside than on the left side.

FIG. 10C illustrates an example in which the LiDAR sensor 810 is rotatedfrom the correct alignment position (e.g., the LiDAR sensor 810 has aroll error). As a result, the IR-transparent portion 830 of thewindshield 840 is rotated with respect to the field of view 910 of theLiDAR sensor 810.

According to some embodiments, the relative position and orientation ofthe IR-transparent portion 830 of the windshield 840 with respect to thefield of view 910 of the LiDAR sensor 810 can be used to calibrate theLiDAR sensor 810. For example, a computing unit can store a referenceimage. The reference can be either acquired by the LiDAR sensor 810while the LiDAR sensor is in the correct alignment position, or can beobtained by simulation. For example, the reference image can be acquiredby the LiDAR sensor just after the LiDAR sensor has been pre-calibratedin a manufacturing facility. When the LiDAR sensor 810 is in normaloperation (either when the vehicle is parked or is driving), thecomputing unit can periodically or continuously compare a current LiDARimage with the reference image. Deviations from the correct alignmentposition of the LiDAR sensor 810 can be made based on the comparison.

In some embodiments, a multi-variable minimization can be performed todetermine a transformation matrix so that the IR-transparent portion 830of the windshield 840 in the current LiDAR image most closely matchesthat in the reference image. The deviations from the correct alignmentposition (e.g., yaw error, roll error, pitch error, ox, 6 y, and 6 z)can then be derived from the transformation matrix. According to variousembodiments, the LiDAR sensor 810 can automatically re-calibrate itself,or provide an alert the vehicle in response to determining that thedeviation from the correct alignment position exceeds a threshold.

Additionally or alternatively, images of some fixed features on thevehicle acquired by the LiDAR sensor 810 can also be used to calibratethe LiDAR sensor 810. FIGS. 10D-10F illustrate some exemplary imagesthat can be acquired by the LiDAR sensor 810. The images show part ofthe hood 1020 of the vehicle and an ornamental feature 1030 (e.g., amanufacturer's logo) attached to the hood 1020. In FIG. 10D, theornamental feature 1030 is approximately laterally centered in the fieldof view 910 of the LiDAR sensor 810. In FIG. 10E, the ornamental feature1030 is shifted to the left of the field of view 910, indicating thatthe LiDAR sensor 810 can be turned toward the right (e.g., has a yawerror). In FIG. 10F, the ornamental feature 1030 is rotated, indicatingthat the LiDAR sensor 810 can be rotated (e.g., has a roll error). Notethat images of the hood 1020 in FIGS. 10E and 10F are also shiftedand/or rotated accordingly. Other exemplary features of the vehicle thatcan be used for calibration include a cover over a LiDAR sensor mountedin the grill, features in a headlamp or tail lamp (e.g, if the LiDARsensor is installed inside the headlamp or tail lamp), and the like.

According to some embodiments, a computing unit can store a referenceimage acquired by the LiDAR sensor while the LiDAR sensor is in acorrect alignment (e.g., an expected alignment) with respect to thevehicle. The reference image can include a first image of a fixedfeature on the vehicle. When the LiDAR sensor 810 is in normal operation(either when the vehicle is parked or is driving), the computing unitcan periodically or continuously compare a current LiDAR image with thereference image. The current LiDAR image includes a second image of thefixed feature of the vehicle. Deviations from the correct alignment canbe determined by comparing the position and orientation of the fixedfeature in the second image to those in the first image.

FIG. 11 shows a simplified flowchart illustrating a method 1100 ofcalibrating a LiDAR sensor mounted on a vehicle using features on thevehicle according to some embodiments.

The method 1100 includes, at 1102, storing a reference three-dimensionalimage acquired by the LiDAR sensor while the LiDAR sensor is in anexpected alignment with respect to the vehicle. The referencethree-dimensional image includes a first image of a fixed feature on thevehicle.

The method 1100 further includes, at 1104, acquiring, using the LiDARsensor, a three-dimensional image including a second image of the fixedfeature.

The method 1100 further includes, at 1106, determining a deviation fromthe expected alignment of the LiDAR sensor with respect to the vehicleby comparing the second image of the fixed feature in thethree-dimensional image to the first image of the fixed feature in thereference three-dimensional image.

In some embodiments, the method 1100 further includes, at 1108,re-calibrating the LiDAR sensor based on the deviation from the expectedalignment of the LiDAR sensor with respect to the vehicle.

In some embodiments, the method 1100 further includes determining atransformation to be applied to the second image of the fixed feature inthe three-dimensional image so as to match the first image of the fixedfeature in the reference three-dimensional image, and re-calibrating theLiDAR sensor based on the transformation.

In some embodiments, the method 1100 further includes determining thatthe deviation from the expected alignment of the LiDAR sensor exceeds athreshold, and providing an alert in response to determining that thedeviation from the expected alignment of the LiDAR sensor exceeds thethreshold.

In some embodiments, the reference three-dimensional image is acquiredby the LiDAR sensor after the LiDAR sensor has been pre-calibrated in amanufacturing facility.

In some embodiments, the fixed feature includes a portion of a hood ofthe vehicle or an object attached to the hood.

In some embodiments, the LiDAR sensor is positioned behind a windshieldof the vehicle, and the fixed feature includes a mask attached to anarea of the windshield that is directly in front of the LiDAR sensor.The mask is configured to block light in an operating wavelength of theLiDAR sensor, and is shaped to block a portion of a field of view of theLiDAR sensor. The mask can have an outer boundary and an inner boundary,and the inner boundary is sized so that the mask encroaches a perimeterof the field of view of the LiDAR sensor.

It should be appreciated that the specific steps illustrated in FIG. 11provide a particular method of calibrating a LiDAR sensor according tosome embodiments. Other sequences of steps may also be performedaccording to alternative embodiments. For example, alternativeembodiments of the present invention may perform the steps outlinedabove in a different order. Moreover, the individual steps illustratedin FIG. 11 may include multiple sub-steps that may be performed invarious sequences as appropriate to the individual step. Furthermore,additional steps may be added and some steps may be removed depending onthe particular applications. One of ordinary skill in the art wouldrecognize many variations, modifications, and alternatives.

C. Dynamic Calibration of LiDAR Sensors Using Road Features

According to some embodiments, a method of dynamic calibration of aLiDAR sensor mounted on a vehicle can use road features as the vehicletravels in a relatively straight section of a road. FIGS. 12A and 12Billustrate an example. A LiDAR sensor 1210 is shown as mounted behindthe front windshield of a vehicle 1220 (the LiDAR sensor 1210 can bemounted in other positions, e.g., on the front bumper). The vehicle 1220is traveling on a straight section of a road 1230. The painted lanemarkings 1240 can be used for dynamically calibrating the position andorientation of the LiDAR sensor 1210 relative to the vehicle 1220 basedon LiDAR images acquired while the vehicle is traveling along the road1230. Using the painted lane markings for calibration can beadvantageous, as they exist almost on all roads. Also, the distancebetween a pair of lane markings 1240 is usually a standard distance. Thelane markings with retroreflective paint may show up clearly in LiDARimages.

As illustrated in FIG. 12A, if the LiDAR sensor 1210 is properly alignedwith respect to the vehicle 1220 (e.g., the LiDAR sensor 1210 is lookingin the same direction as the vehicle 1220 is heading), the pair of lanemarkings 1240 may appear to pass equally on each side of the vehiclepath 1250. As illustrated in FIG. 12B, if the LiDAR sensor 1210 ismis-aligned with respect to the vehicle 1220 (e.g., the LiDAR sensor1210 is looking to the left with respect to the longitudinal axis of thevehicle 1220), then the pair of lane markings 1240 may appear to passthe vehicle 1220 asymmetrically with respect to the vehicle path 1250.For example, the vehicle path 1250 can appear as moving closer to thelane marking 1240 a on the driver side than to the lane marking 1240 bon the passenger side. Thus, by analyzing the LiDAR images of the lanemarkings 1240 with respect to the vehicle path 1250, the amount ofmis-alignment (e.g., the yaw error) of the LiDAR sensor 1210 withrespect to the vehicle 1220 can be estimated.

FIGS. 13A-13C illustrate some more examples of using lane makings fordynamic calibration of a LiDAR sensor mounted on a vehicle according tosome embodiments. As illustrated in FIG. 13A, if the alignment of theLiDAR sensor 1210 has a pitch error, the lane markings 1240 may appearto be tilted with respect to the vehicle path (assuming that the road isrelatively level).

As illustrated in FIG. 13B, if the alignment of the LiDAR sensor 1210has a roll error, the lane marking 1240 a on one side of the vehicle1220 (e.g., the driver side) may appear to be higher than the lanemarking 1240 b on the other side of the vehicle 1220 (e.g., thepassenger side).

As illustrated in FIG. 13C, if the alignment of the LiDAR sensor 1210has a Z error (in the vertical direction), the height of the lanemarkings 1240 may be offset from the vehicle path. Translational errorsin the other two orthogonal directions (e.g., X error and Y error) canalso be detected by monitoring the motions of objects such as lanemarkings (vs. expected motions) as the vehicle 1220 travels forward.

Thus, by analyzing the spatial relationship between the images of thelane markings 1240 (or other road features) and the vehicle path,various rotational and translational mis-alignments of the LiDAR sensor1210 with respect to the vehicle 1220 can be detected and estimated. Ifa LiDAR sensor is mounted on the side or the rear of the vehicle,similar calibration procedures can be used, with appropriatemodification and mathematical transformations to account for thedifferent view angles.

According to some embodiments, measurements can be repeated a number oftimes, and the results can be averaged to account for, for example,roadway irregularities, poorly painted lane markings, curves, dips,potholes, and the like. Outliers can be discarded. Outliers can resultfrom, for example, lane changes and other driving irregularities,obstruction of view of the lane markings by other vehicles, and thelike.

According to some embodiments, other information can be used to augmentthe quality and reliability of the calibration data. For example, datafrom the vehicle steering sensor, global navigation satellite systems(e.g., GPS) data, and inertial measurement unit (IMU) data, and thelike, can be used to make sure the vehicle is not turning. Map data canbe used to select good road sections for calibration, where the road isstraight, level, and the lane markings are fresh and properly spaced.Other road features such as curbs, guardrails, and road signs may alsobe used as input to the calibration algorithm.

FIG. 14 shows a simplified flowchart illustrating a method 1400 ofcalibrating a LiDAR sensor mounted on a vehicle using road featuresaccording to some embodiments.

The method 1400 includes, at 1402, acquiring, using the LiDAR sensorwhile the vehicle is traveling on a road with fixed road features, oneor more three-dimensional images. Each of the one or morethree-dimensional images includes images of the road features.

The method 1400 further includes, at 1404, analyzing a spatialrelationship between the images of the road features in the one or morethree-dimensional images and an orientation of a field of view of theLiDAR sensor.

The method 1400 further includes, at 1406, determining a deviation froman expected alignment of the LiDAR sensor with respect to the vehiclebased on the spatial relationship between the images of the roadfeatures and the field of view of the LiDAR sensor.

In some embodiments, the method 1400 further includes, at 1408,re-calibrating the LiDAR sensor based on the deviation from the expectedalignment of the LiDAR sensor with respect to the vehicle.

In some embodiments, the method 1400 further includes determining thatthe deviation from the expected alignment of the LiDAR sensor exceeds athreshold, and providing an alert in response to determining that thedeviation from the expected alignment of the LiDAR sensor exceeds thethreshold.

In some embodiments, the road features include one or more pairs of lanemarkings on either side of the vehicle. Analyzing the spatialrelationship can include determining a pitch angle between one pair oflane markings of the one or more pairs of lane markings and the field ofview of the LiDAR sensor; and determining the deviation from theexpected alignment of the LiDAR sensor can include determining a pitcherror of the LiDAR sensor based on the pitch angle. In some embodiments,the one or more pairs of lane markings can include a first lane markingon a driver side of the vehicle and a second lane marking on a passengerside of the vehicle; analyzing the spatial relationship can includedetermining a height difference between the first lane marking and thesecond lane marking; and determining the deviation from the expectedalignment of the LiDAR sensor can include determining a roll error ofthe LiDAR sensor based on the height difference.

In some embodiments, the one or more three-dimensional images caninclude a plurality of three-dimensional images acquired by the LiDARsensor over an interval of time as the vehicle is traveling on the roadfor an interval of distance. The road can be substantially straight andlevel over the interval of distance. In some embodiments, the method1400 further includes determining a path of the vehicle over theinterval of distance, and comparing the path of the vehicle to paths ofthe road features from the plurality of three-dimensional images. Insome embodiments, the one or more pairs of lane markings include a firstlane marking on a driver side of the vehicle and a second lane markingon a passenger side of the vehicle; analyzing the spatial relationshipcan include determining an amount of lateral asymmetry between the firstlane marking from the path of the vehicle and the second lane markingfrom the path of the vehicle; and determining the deviation from theexpected alignment of the LiDAR sensor can include determining a yawerror of the LiDAR sensor based on the amount of lateral asymmetry. Insome embodiments, analyzing the spatial relationship can includedetermining a height difference between one pair of lane markings andthe path of the vehicle; and determining the deviation from the expectedalignment of the LiDAR sensor can include determining a vertical errorof the LiDAR sensor based on the height difference.

It should be appreciated that the specific steps illustrated in FIG. 14provide a particular method of calibrating a LiDAR sensor according tosome embodiments. Other sequences of steps may also be performedaccording to alternative embodiments. For example, alternativeembodiments of the present invention may perform the steps outlinedabove in a different order. Moreover, the individual steps illustratedin FIG. 14 may include multiple sub-steps that may be performed invarious sequences as appropriate to the individual step. Furthermore,additional steps may be added and some steps may be removed depending onthe particular applications. One of ordinary skill in the art wouldrecognize many variations, modifications, and alternatives.

It is also understood that the examples and embodiments described hereinare for illustrative purposes only and that various modifications orchanges in light thereof will be suggested to persons skilled in the artand are to be included within the spirit and purview of this applicationand scope of the appended claims.

What is claimed is:
 1. A method of calibrating a LiDAR sensor mounted ona vehicle, the method comprising: storing a reference three-dimensionalimage acquired by the LiDAR sensor while the LiDAR sensor is in anexpected alignment with respect to the vehicle, the referencethree-dimensional image including a first image of a fixed feature onthe vehicle; acquiring, using the LiDAR sensor, a three-dimensionalimage including a second image of the fixed feature; and determining adeviation from the expected alignment of the LiDAR sensor with respectto the vehicle by comparing the second image of the fixed feature in thethree-dimensional image to the first image of the fixed feature in thereference three-dimensional image.
 2. The method of claim 1 furthercomprising: re-calibrating the LiDAR sensor based on the deviation fromthe expected alignment of the LiDAR sensor with respect to the vehicle.3. The method of claim 1 further comprising: determining atransformation to be applied to the second image of the fixed feature inthe three-dimensional image so as to match the first image of the fixedfeature in the reference three-dimensional image; and re-calibrating theLiDAR sensor based on the transformation.
 4. The method of claim 1further comprising: determining that the deviation from the expectedalignment of the LiDAR sensor exceeds a threshold; and providing analert in response to determining that the deviation from the expectedalignment of the LiDAR sensor exceeds the threshold.
 5. The method ofclaim 1, wherein the reference three-dimensional image is acquired bythe LiDAR sensor after the LiDAR sensor has been pre-calibrated in amanufacturing facility.
 6. The method of claim 1 wherein the deviationfrom the expected alignment of the LiDAR sensor includes one or more ofa yaw deviation, a roll deviation, a pitch deviation, and translationaldeviations along three orthogonal axes.
 7. The method of claim 1 whereinthe fixed feature comprises a portion of a hood of the vehicle or anobject attached to the hood.
 8. The method of claim 1 wherein: the LiDARsensor is positioned behind a windshield of the vehicle; and the fixedfeature comprises a mask attached to an area of the windshield that isdirectly in front of the LiDAR sensor, wherein the mask is configured toblock light in an operating wavelength of the LiDAR sensor and is shapedto block a portion of a field of view of the LiDAR sensor.
 9. The methodof claim 8 wherein the mask has an outer boundary and an inner boundary,the inner boundary being sized so that the mask encroaches a perimeterof the field of view of the LiDAR sensor.
 10. A method of calibrating aLiDAR sensor mounted on a vehicle, the method comprising: acquiring,using the LiDAR sensor while the vehicle is traveling on a road withfixed road features, one or more three-dimensional images, each of theone or more three-dimensional images including images of the roadfeatures; analyzing a spatial relationship between the images of theroad features in the one or more three-dimensional images and anorientation of a field of view of the LiDAR sensor; and determining adeviation from an expected alignment of the LiDAR sensor with respect tothe vehicle based on the spatial relationship between the images of theroad features and the field of view of the LiDAR sensor.
 11. The methodof claim 10 further comprising: re-calibrating the LiDAR sensor based onthe deviation from the expected alignment of the LiDAR sensor withrespect to the vehicle.
 12. The method of claim 10 further comprising:determining that the deviation from the expected alignment of the LiDARsensor exceeds a threshold; and providing an alert in response todetermining that the deviation from the expected alignment of the LiDARsensor exceeds the threshold.
 13. The method of claim 10 wherein theroad features comprise one or more pairs of lane markings on either sideof the vehicle.
 14. The method of claim 13 wherein: analyzing thespatial relationship comprises determining a pitch angle between onepair of lane markings of the one or more pairs of lane markings and thefield of view of the LiDAR sensor; and determining the deviation fromthe expected alignment of the LiDAR sensor comprises determining a pitcherror of the LiDAR sensor based on the pitch angle.
 15. The method ofclaim 13 wherein: the one or more pairs of lane markings include a firstlane marking on a driver side of the vehicle and a second lane markingon a passenger side of the vehicle; analyzing the spatial relationshipcomprises determining a height difference between the first lane markingand the second lane marking; and determining the deviation from theexpected alignment of the LiDAR sensor comprises determining a rollerror of the LiDAR sensor based on the height difference.
 16. The methodof claim 13 wherein the one or more three-dimensional images comprise aplurality of three-dimensional images acquired by the LiDAR sensor overan interval of time as the vehicle is traveling on the road for aninterval of distance.
 17. The method of claim 16 wherein the road issubstantially straight and level over the interval of distance.
 18. Themethod of claim 16 further comprising: determining a path of the vehicleover the interval of distance; and comparing the path of the vehicle topaths of the road features from the plurality of three-dimensionalimages.
 19. The method of claim 18 wherein: the one or more pairs oflane markings include a first lane marking on a driver side of thevehicle and a second lane marking on a passenger side of the vehicle;analyzing the spatial relationship comprises determining an amount oflateral asymmetry between the first lane marking from the path of thevehicle and the second lane marking from the path of the vehicle; anddetermining the deviation from the expected alignment of the LiDARsensor comprises determining a yaw error of the LiDAR sensor based onthe amount of lateral asymmetry.
 20. The method of claim 18 wherein:analyzing the spatial relationship comprises determining a heightdifference between one pair of lane markings and the path of thevehicle; and determining the deviation from the expected alignment ofthe LiDAR sensor comprises determining a vertical error of the LiDARsensor based on the height difference.